線形回帰の場合，残差の平方和がδの二乗に従うか？
・傾きを定数とした場合の最小二乗
線形の式，
まずは，\(\Large \displaystyle y_i =a_0 + a_1 x_i \)，からスタートします．
傾きを固定したので，傾きの固定した値によって，切片，a0，がどのような最適解になるかを調べます．
\(\Large \displaystyle \sum_{i=1}^n \left( y_i - a_0 - a_1 x_i \right)^2 \)
が最小になればいいので，
\(\Large \displaystyle \frac{ \partial}{ \partial a_0} \sum_{i=1}^n \left( y_i - a_0 - a_1 x_i \right)^2 =0 \)
となる，a0，を求めます．
\(\Large \displaystyle = \frac{ \partial}{ \partial a_0} \left[ \sum_{i=1}^n y_i^2
+\sum_{i=1}^n a_0^2 +\sum_{i=1}^n ( b x_i)^2
- 2 \sum_{i=1}^n
y_i a_0 -2 \sum_{i=1}^n a_1 x_i y_i +2 \sum_{i=1}^n a_0 a_1 xi
\right] \)
a0，が含まれていない項は微分で０となるので，
\(\Large \displaystyle = \frac{ \partial}{ \partial a_0} \left[
n a_0^2 - 2 a_0 \sum_{i=1}^n
y_i +2 a_0 a_1\sum_{i=1}^n xi
\right] \)
\(\Large \displaystyle = 2 n a_0 - 2 \sum_{i=1}^n y_i +2 a_1\sum_{i=1}^n xi =0 \)
\(\Large \displaystyle a_{0 a_1} = \frac{1}{n} \left( \sum_{i=1}^n
y_i - a_1\sum_{i=1}^n xi \right)
= \bar{y} - a_1 \bar{x}
\)
となります．
ここで，a1，を変化させたときの残差は，
\(\Large \displaystyle \sum_{i=1}^n \left( y_i - a_{0 a_1} - a_1 x_i \right)^2 \)
\(\Large \displaystyle = \sum_{i=1}^n \left\{ y_i - (\bar{y} - a_1 \bar{x}) - a_1 x_i \right\}^2 \)
\(\Large \displaystyle = \sum_{i=1}^n \left\{ (y_i - \bar{y}) - a_1 (x_i - \bar{x}) \right\}^2 \)
ここでa1，を推定値からの偏差，δ，で表すと，
\(\Large \displaystyle a_1 = \hat{a_1} + \delta \)
となるので，
\(\Large \displaystyle = \sum_{i=1}^n \left\{ (y_i - \bar{y}) - (\hat{a_1} + \delta) (x_i - \bar{x}) \right\}^2 \)
\(\Large \displaystyle = \sum_{i=1}^n \left\{ (x_i - \bar{x}) \delta
+ (y_i - \bar{y}) - \hat{a_1} (x_i - \bar{x}) \right\}^2 \)
と整理できます．
次に，δ2，δ1，δ0，と分けて考えていきましょう．
・δ2
これは簡単に，
\(\Large \displaystyle \sum_{i=1}^n (x_i - \bar{x})^2 \)
となります．
・δ1
\(\Large \displaystyle 2 \sum_{i=1}^n \left[ (x_i - \bar{x}) \left\{ (y_i - \bar{y}) - \hat{a_1} (x_i - \bar{x}) \right\} \right] \)
\(\Large \displaystyle = 2 \sum_{i=1}^n \left[ (x_i - \bar{x}) (y_i - \bar{y}) - \hat{a_1} (x_i - \bar{x})^2 \right] \)
\(\Large \displaystyle = 2 \sum_{i=1}^n \left[ x_i y_i -x_i \bar{y} - \bar{x} y_i +\bar{x} \bar{y}
- \hat{a_1} x_i^2 + 2 \hat{a_1} x_i \bar{x} - \hat{a_1} \bar{x}^2 \right] \)
\(\Large \displaystyle = 2 n \left[ \overline{xy} -\bar{x} \bar{y} - \bar{x} \bar{y} +\bar{x} \bar{y}
- \hat{a_1} \bar{x^2} + 2 \hat{a_1} \bar{x}^2- \hat{a_1} \bar{x}^2 \right] \)
\(\Large \displaystyle = 2 n \left[ \overline{xy} -\bar{x} \bar{y}
- \hat{a_1} \bar{x^2} + \hat{a_1} \bar{x}^2 \right] \)
\(\Large \displaystyle = 2 n \left[ \overline{xy} -\bar{x} \bar{y}
- \hat{a_1} (\bar{x^2} - \bar{x}^2) \right] \)
となります．ここで，
\(\Large \displaystyle \bar{x^2} , \bar{x}^2 \)
の違いに注意です．ここで，
\(\Large \displaystyle \hat{a_1} = \frac{ \displaystyle \sum_{i=1}^{n} \left(x_i - \bar{x} \right) \left(y_i - \bar{y} \right)}{ \displaystyle \sum_{i=1}^{n} \left( x_i - \bar{x} \right)^2} \)
\(\Large \displaystyle = \frac{ \displaystyle \sum_{i=1}^{n} \left(x_i y_i -x_i \bar{y} - \bar{x} y_i +\bar{x} \bar{y}\right)}{ \displaystyle \sum_{i=1}^{n} \left( x_i^2 - 2 x_i \bar{x} +\bar{x}^2 \right)} \)
\(\Large \displaystyle = \frac{ n \left(\overline{xy} -\bar{x} \bar{y} - \bar{x} \bar{y} +\bar{x} \bar{y}\right)}
{ n \left( \bar{x^2} - 2 \bar{x}^2 +\bar{x}^2 \right)} \)
\(\Large \displaystyle = \frac{ \overline{xy} -\bar{x} \bar{y} }
{ \bar{x^2} - \bar{x}^2 } \)
ですので，先程の式に代入すると，
\(\Large \displaystyle = 2 n \left[ \overline{xy} -\bar{x} \bar{y}
- \hat{a_1} (\bar{x^2} - \bar{x}^2) \right] \)
\(\Large \displaystyle = 2 n \left[ \overline{xy} -\bar{x} \bar{y}
- \frac{ \overline{xy} -\bar{x} \bar{y} }
{ \bar{x^2} - \bar{x}^2 } (\bar{x^2} - \bar{x}^2) \right] \)
\(\Large \displaystyle = 2 n \left[ \overline{xy} -\bar{x} \bar{y}
- ( \overline{xy} -\bar{x} \bar{y}) \right] = 0 \)
となって，δ1，の項は０となります．
・δ0
\(\Large \displaystyle \sum_{i=1}^n \left[\left\{ (y_i - \bar{y}) - \hat{a_1} (x_i - \bar{x}) \right\}^2 \right] \)
\(\Large \displaystyle = \sum_{i=1}^n \left[ (y_i - \bar{y})^2 - 2 \hat{a_1} (x_i - \bar{x}) (y_i - \bar{y}) + \hat{a_1}^2 (x_i - \bar{x})^2 \right] \)
\(\Large \displaystyle = \sum_{i=1}^n [ y_i^2 - 2 \bar{y} y_i + \bar{y}^2 \)
\(\Large \displaystyle \hspace{40 pt} - 2 \hat{a_1} x_i y_i +2 \hat{a_1} x_i \bar{y} +2 \hat{a_1} \bar{x} y_i - 2 \hat{a_1} \bar{x} \bar{y}
\)
\(\Large \displaystyle \hspace{40 pt} + \hat{a_1}^2 x_i^2 - 2 \hat{a_1}^2 \bar{x} x_i + \hat{a_1}^2 \bar{x}^2] \)
\(\Large \displaystyle = n [ \bar{y^2} - 2 \bar{y}^2 + \bar{y}^2 \)
\(\Large \displaystyle \hspace{40 pt} - 2 \hat{a_1} \overline{x y} +2 \hat{a_1} \bar{x} \bar{y} +2 \hat{a_1} \bar{x} \bar{y} - 2 \hat{a_1} \bar{x} \bar{y}
\)
\(\Large \displaystyle \hspace{40 pt} + \hat{a_1}^2 \bar{x^2} - 2 \hat{a_1}^2 \bar{x}^2 + \hat{a_1}^2 \bar{x}^2] \)
となります．同じ項目を色分けすると，
\(\Large \displaystyle = n [ \bar{y^2} \color{red}{- 2 \bar{y}^2 + \bar{y}^2} \)
\(\Large \displaystyle \hspace{40 pt} - 2 \hat{a_1} \overline{x y} +\color{blue}{2 \hat{a_1} \bar{x} \bar{y} +2 \hat{a_1} \bar{x} \bar{y} - 2 \hat{a_1} \bar{x} \bar{y} }
\)
\(\Large \displaystyle \hspace{40 pt} + \hat{a_1}^2 \bar{x^2} \color{purple}{- 2 \hat{a_1}^2 \bar{x}^2 + \hat{a_1}^2 \bar{x}^2]} \)
となるので，整理すると，
\(\Large \displaystyle = n \left[ \bar{y^2} - \bar{y}^2 - 2 \hat{a_1} \overline{x y} +2 \hat{a_1} \bar{x} \bar{y} + \hat{a_1}^2 \bar{x^2} - \hat{a_1}^2 \bar{x}^2 \right] \)
\(\Large \displaystyle = n \left[ \bar{y^2} - \bar{y}^2 - 2 \hat{a_1} (\overline{x y} - \bar{x} \bar{y}) + \hat{a_1}^2 (\bar{x^2} - \bar{x}^2) \right] \)
ここから，\(\Large\displaystyle \hat{a_1} = \frac{ \overline{x y} - \bar{x} \bar{y} }
{ \displaystyle
\overline{x^2}- \bar{x}^2}
\)，
\(\Large \displaystyle = n \left[ \bar{y^2} - \bar{y}^2
- 2 \frac{ \overline{x y} - \bar{x} \bar{y} }
{ \displaystyle
\overline{x^2}- \bar{x}^2} (\overline{x y} - \bar{x} \bar{y})
+ \left( \frac{ \overline{x y} - \bar{x} \bar{y} }
{ \displaystyle
\overline{x^2}- \bar{x}^2} \right)^2 (\bar{x^2} - \bar{x}^2) \right] \)
\(\Large \displaystyle = n \left[ \bar{y^2} - \bar{y}^2
- 2 \frac{ (\overline{x y} - \bar{x} \bar{y})^2 }
{ \displaystyle
\overline{x^2}- \bar{x}^2}
+ \frac{ (\overline{x y} - \bar{x} \bar{y})^2 }
{ \displaystyle
\overline{x^2}- \bar{x}^2} \right] \)
\(\Large \displaystyle = n \left[ \bar{y^2} - \bar{y}^2
- \frac{ (\overline{x y} - \bar{x} \bar{y})^2 }
{ \displaystyle
\overline{x^2}- \bar{x}^2}
\right] \)
この結果は，すなわち，前ページ，の，
\(\Large \displaystyle Se = \sum_{i=1}^{n} \left( y_i -\hat{a_0} - \hat{a_1} x_i \right)^2 \)
と同じになるので，まとめると，a1，を変化させたときの残差は，
\(\Large \displaystyle \sum_{i=1}^n \left( y_i - a_{0b} - a_1 x_i \right)^2 = Se + \sum_{i=1}^n (x_i - \bar{x})^2 \delta^2 \)
となります．
ここで，これらのサイト，サイト，サイトでお示ししたように，\(\Large \displaystyle \hat{a_1} \)の分散は，
\(\Large \displaystyle V \left[ \hat{a_1} \right] = \sigma^2 \sum_{i=1}^{n} \omega_i^2
= \frac{\sigma^2 }{\sum_{i=1}^{n} \left( x_i - \bar{x} \right)^2} \)
\(\Large \displaystyle \sigma^2 = \frac{E \left[ \sum_{i=1}^n \hat{u_i}^2 \right]}{n-2} = s^2 \)
\(\Large \displaystyle y_i = \hat{a_0} + \hat{a_1} x_i + \hat{u_i} \)
から，
\(\Large \displaystyle \hat{u_i} = y_i - \hat{a_0} - \hat{a_1} x_i \)
\(\Large \displaystyle \sigma^2 = \frac{E \left[ \sum_{i=1}^n \hat{u_i}^2 \right]}{n-2} =\frac{ \sum_{i=1}^n (y_i - \hat{a_0} - \hat{a_1} x_i)^2 }{n-2} \)
\(\Large \displaystyle V \left[ \hat{a_1} \right]
= \frac{\sigma^2 }{\sum_{i=1}^{n} \left( x_i - \bar{x} \right)^2}
=
\frac{\sum_{i=1}^n (y_i - \hat{a_0} - \hat{a_1} x_i)^2 }{ (n-2) \sum_{i=1}^{n} \left( x_i - \bar{x} \right)^2} \)
となります．ここでシフト量，δの二乗が,\(\Large \displaystyle V \left[ \hat{a_1} \right] \)，の場合は，
\(\Large \displaystyle \sum_{i=1}^n \left( y_i - a_{0b} - a_1 x_i \right)^2 = Se + \sum_{i=1}^n (x_i - \bar{x})^2 \delta^2 \)
\(\Large \displaystyle = Se + \sum_{i=1}^n (x_i - \bar{x})^2 \frac{\sum_{i=1}^n (y_i - \hat{a_0} - \hat{a_1} x_i)^2 }{ (n-2) \sum_{i=1}^{n} \left( x_i - \bar{x} \right)^2} \)
\(\Large \displaystyle = Se + \frac{\sum_{i=1}^n (y_i - \hat{a_0} - \hat{a_1} x_i)^2 }{ n-2} \)
となり，第二項が，ここ，で説明した，
\(\Large \displaystyle Ve = \frac{1}{n-2} \sum_{i=1}^{n} \left(y_i -\hat{a_0} - \hat{a_1} x_i \right)^2 = \frac{Se}{n-2} \)
と等しくなり，結果として，
\(\Large \displaystyle S_{SE} = Se + Ve \)
が成り立つことになります．